Unity - Android Share Texture

发表于   |   分类于   |   2,217 字   |   11 分钟   |  

Unity - Android Share Texture

需求来源

原始的需求是打算在Unity中申请一个Texture，但是Texture的source部分是由Android提供的。

应用的场景：在Unity中显示一个Android App的画面。

技术点

  • [x] Unity & Android Share Texture
  • [x] Android VirtualDisplay
  • [x] OpenGLES
  • [x] FBO

Pipeline 流程图

sequenceDiagram

# init flow
Note left of Unity: 1.Init流程
Unity->>AndroidPlugin:initVirtualDisplay(unityTextureId)
activate AndroidPlugin

activate AndroidPlugin
AndroidPlugin->>AndroidPlugin:createOESTexture
deactivate AndroidPlugin
Note right of AndroidPlugin: 创建OESTexture


AndroidPlugin->>SurfaceTexture:new SurfaceTexture(OESTextureId)
activate SurfaceTexture
Note right of SurfaceTexture: 通过OESTexture创建SurfaceTexture
SurfaceTexture-->>AndroidPlugin:mSurfaceTexture
deactivate SurfaceTexture

AndroidPlugin->>Surface:new Surface(mSurfaceTexture)
activate Surface
Note right of Surface: 通过SurfaceTexture创建Surface
Surface-->>AndroidPlugin:mSurface
deactivate Surface

AndroidPlugin ->> FBOHelper: new FBO(unityTextureId,OESTextureId)
activate FBOHelper
Note right of FBOHelper: 构造FBOHelper类
activate FBOHelper
FBOHelper ->> FBOHelper: genFBO
deactivate FBOHelper
FBOHelper -->> AndroidPlugin: mFBOHelper
deactivate FBOHelper

AndroidPlugin ->> VirtualDisplay:createVirtualDisplay(mSurface)
activate VirtualDisplay
Note right of VirtualDisplay: 通过mSruface创建出VirtualDisplay
VirtualDisplay -->> AndroidPlugin:mVirtualDisplay
deactivate VirtualDisplay

AndroidPlugin-->>Unity: done with initVirtualDisplay
deactivate AndroidPlugin
Note left of Unity: 完成VirtualDisplay初始化

# VirtualDisplay updated
Unity -->> VirtualDisplay: Update VirtualDisplay
Note left of Unity: 2.VirtualDisplay画面更新流程
Unity ->> App: startActivity(display id)
activate Unity
activate App
App->>App: SurfaceControl.setLayerStack(virtual display id)
App -->> Unity: end startActivity
deactivate App
deactivate Unity
Note right of App: App画面有更新

App->>App: invalidate()
App-->>VirtualDisplay: 
VirtualDisplay ->> Surface : onFrameAvailable
activate Surface
Note right of Surface: VirtualDisplay画面有更新时
App->>App: ......
App-->>VirtualDisplay: 
Note right of Surface:实际Surface数据会刷新
Surface -->> VirtualDisplay : 
deactivate Surface

App->>App: invalidate()
App-->>VirtualDisplay: 
VirtualDisplay ->> Surface : onFrameAvailable
activate Surface
Surface -->> VirtualDisplay : 
deactivate Surface

VirtualDisplay ->> Surface : onFrameAvailable
activate Surface
Surface -->> VirtualDisplay : 
deactivate Surface
App->>App: invalidate()
App-->>VirtualDisplay: 
Note right of Surface: VirtualDisplay的更新如上述循环
Note left of Unity: VirtualDisplay画面更新流程
Unity --> VirtualDisplay: 

# draw
Note left of Unity: 3.Unity画面更新流程
Unity->>AndroidPlugin:isFrameUpdated()
activate AndroidPlugin
AndroidPlugin-->>Unity:return True
deactivate AndroidPlugin

Unity->>AndroidPlugin:draw()
activate AndroidPlugin
AndroidPlugin ->> SurfaceTexture: updateTexImage
activate SurfaceTexture
SurfaceTexture -->> AndroidPlugin: 
deactivate SurfaceTexture
Note left of SurfaceTexture: 更新OESTexture
AndroidPlugin ->> FBOHelper: draw()
activate FBOHelper
activate FBOHelper
FBOHelper -> FBOHelper: glBindFramebuffer
Note right of FBOHelper: 激活FBO
deactivate FBOHelper
activate FBOHelper
FBOHelper -> FBOHelper:glFramebufferTexture2D
Note right of FBOHelper: 绑定UnityTexture到FBO
deactivate FBOHelper
activate FBOHelper
FBOHelper -> FBOHelper:glBindTexture
Note right of FBOHelper: 绑定OESTexture到当前激活纹理单元，进行绘制
deactivate FBOHelper
FBOHelper-->>AndroidPlugin:done draw()
deactivate FBOHelper
Note right of AndroidPlugin: 完成了OESTexutre对2DTexture的输出，更新了画面
AndroidPlugin-->>Unity:done draw()
deactivate AndroidPlugin
Note left of Unity: Unity画面更新完成

流程描述

  • 把Android端创建的VirtualDisplay上的内容绘制到这块纹理上
    • VirtualDisplay内容的绘制流程：
      1. VirtualDisplay画面的改变会刷新到绑定的Surface上
        • VirtualDisplay -> Surface（CPU端数据处理）
      2. 更新Surface绑定的SurfaceTexture对象的纹理
        • Surface -> SurfaceTexture（CPU内存数据更新到GPU端）
          • SurfaceTexture.updateTexImage方法
      3. SurfaceTexture绑定的纹理是GL_TEXTURE_EXTERNAL_OES
      4. 使用FBO（Frame Buffer Object）关联Unity的GL_TEXTURE_2D
      5. 把GL_TEXTURE_EXTERNAL_OES的内容绘制到FBO
        • SurfaceTexture -> FBO（GPU端数据处理）
      6. 最终GL_TEXTURE_EXTERNAL_OES的内容就同步到了GL_TEXTURE_2D
        • FBO -> GL_TEXTURE_2D（GPU端数据处理）

代码实现

根据流程图的实现，整个代码的解析可以分成4个部分：

  • 创建OESTexture
  • 根据OESTexture创建SurfaceTexture
  • 构建FBOHelper类，关联Unity 2DTexture
  • 绘制流程Draw

创建OESTexture

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
    public static int createOESTextureID() {
        int[] id = new int[1];
        //生成纹理，GLES30.glGenTextures(number, outIds, offset);
        GLES30.glGenTextures(1, id, 0);
        //将OESTexture Object绑定到当前激活的texture的texture target上
        GLES30.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, id[0]);
        //设置OESTexture的参数
        GLES30.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES30.GL_TEXTURE_MIN_FILTER, GLES30.GL_LINEAR_MIPMAP_LINEAR);
        GLES30.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES30.GL_TEXTURE_MAG_FILTER, GLES30.GL_LINEAR);
        GLES30.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES30.GL_TEXTURE_WRAP_S, GLES30.GL_CLAMP_TO_EDGE);
        GLES30.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES30.GL_TEXTURE_WRAP_T, GLES30.GL_CLAMP_TO_EDGE);
        //生成OESTexture的Mipmap
        GLES30.glGenerateMipmap(GLES11Ext.GL_TEXTURE_EXTERNAL_OES);
        return id[0];
    }
mCreateOESTextureID = createOESTextureID()

创建SurfaceTexture

1
2
3
   // 通过OESTexture来创建SurfaceTexture
mSurfaceTexture = new SurfaceTexture(mCreateOESTextureID);
   mSurface = new Surface(mSurfaceTexture);

其中可以特别参考一下SurfaceTexture的开头注释，其中有两个点需要注意的：

  • 第一个点是SurfaceTexture用到的是OESTexture，在使用Shader的时候需要额外声明：

    • "#extension GL_OES_EGL_image_external : require"

    The texture object uses the GL_TEXTURE_EXTERNAL_OES texture target, which is defined by the GL_OES_EGL_image_external OpenGL ES extension. This limits how the texture may be used. Each time the texture is bound it must be bound to the GL_TEXTURE_EXTERNAL_OES target rather than the GL_TEXTURE_2D target. Additionally, any OpenGL ES 2.0 shader that samples from the texture must declare its use of this extension using, for example, an “#extension GL_OES_EGL_image_external : require” directive. Such shaders must also access the texture using the samplerExternalOES GLSL sampler type.

  • 第二个点是对应的OESTexture的更新是通过调用updateTexImage来触发的

    When updateTexImage is called, the contents of the texture object specified when the SurfaceTexture was created are updated to contain the most recent image from the imagestream. This may cause some frames of the stream to be skipped.

构建FBOHelper类

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
  	// fragment Shader，其中in参数为v_TexCoord
    private static final String fragmentShaderCode =
        "#version 300 es  \n" +
        "#extension GL_OES_EGL_image_external_essl3 : require    \n" +
        "#extension GL_OES_EGL_image_external : require          \n" +
        "precision mediump float;                                \n" +
        "in vec2 v_TexCoord;                                     \n" +
        "out vec4 fragColor;                                     \n" +
        "uniform samplerExternalOES s_Texture;                   \n" +
        "void main() {                                           \n" +
        "   fragColor = texture(s_Texture, v_TexCoord);          \n" +
        "}                                                       \n";
    // vertex Shader，其中in参数为a_Position和a_TexCoord
    private static final String vertexShaderCode =
        "#version 300 es                       \n" +
        "in vec4 a_Position;           \n" +
        "in vec2 a_TexCoord;           \n" +
        "out vec2 v_TexCoord;          \n" +
        "void main() {                 \n" +
        "   gl_Position = a_Position;  \n" +
        "   v_TexCoord = a_TexCoord;   \n" +
        "}                             \n";
//因此，fragment和vertex shader一共需要3个参数，其中v_TexCoord是vetex shader的output，但又是fragment shader的input
    private final float[] textureData = {0.0f, 0.0f, 1.0f, 0.0f, 0.0f, 1.0f, 1.0f, 1.0f};
    private final float[] vertexData = {-1.0f, 1.0f, 1.0f, 1.0f, -1.0f, -1.0f, 1.0f, -1.0f}; 
FBOTexture(int unityTextureNativePtr, int oesTextureId) {
        //记录unity传入的texture id，一般为2DTexture
        //记录OESTexture的id
        unityTextureId = unityTextureNativePtr;
        oesTextureId = oesTextureId;
        //创建 fbo
        int[] fboId = new int[1];
        GLES30.glGenFramebuffers(1, fboId, 0);
        mFBO = fboId[0];
        //创建Vertex Buffer Object
        int[] textureIds = new int[2];
        GLES30.glGenBuffers(2, textureIds, 0);
        vertexVBO = textureIds[0];
        textureVBO = textureIds[1];
    
        //创建 shader
        int buildProgram = FBOUtils.buildProgram(vertexShaderCode, fragmentShaderCode);
        shaderProgram = buildProgram;
        GLES30.glUseProgram(buildProgram);
        //获取到shader中的in参数
        a_Position = GLES30.glGetAttribLocation(shaderProgram, "a_Position");
        a_TexCoord = GLES30.glGetAttribLocation(shaderProgram, "a_TexCoord");
        s_Texture = GLES30.glGetUniformLocation(shaderProgram, "s_Texture");
    }

整个程序完成了：

  • 保存Unity创建的Texture
  • 保存了OESTexture
  • 创建了FBO
  • 创建了VBO
  • 创建了Fragment Shader和VertexShader
  • 获取了Shader中的参数

绘制流程Draw

上述的种种都是为了在最后进行绘制，当有数据来临的时候，根据之前对SurfaceTexture的理解，我们需要完成最后的OES->2DTexture的数据填充。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
public void updateTexture() {
    mSurfaceTexture.updateTexImage();
    mFBOTexture.draw();
}
FBOTexture::draw()
// 在FBOTexture中已经完成的初始化：
//	mFBO = fboId[0];
    
public void draw() {
    //设置操作的视口为全屏幕，这里的width和height可以是virtual display的实际大小
    GLES30.glViewport(0, 0, width, height);
    //清屏
    GLES30.glClearColor(1.0f, 1.0f, 1.0f, 0.0f);
    GLES30.glClear(GLES30.GL_COLOR_BUFFER_BIT | GLES30.GL_DEPTH_BUFFER_BIT);
    
    //绑定mFBO的texture为当前FrameBuffer激活的texture
    GLES30.glBindFramebuffer(GLES30.GL_FRAMEBUFFER, mFBO);
    //绑定unity texuture附加到FrameBuffer Object，也就是画在FBO上的数据会变成画在unity texture
    GLES30.glFramebufferTexture2D(GLES30.GL_FRAMEBUFFER,
                                  GLES30.GL_COLOR_ATTACHMENT0, 
                                  GLES30.GL_TEXTURE_2D, 
                                  unityTextureId, 0);
    //启用shader
    GLES30.glUseProgram(shaderProgram);
    //这一段到下面没有看懂？？？
    GLES30.glBindBuffer(GLES30.GL_ARRAY_BUFFER, vertexVBO);
    GLES30.glBufferData(GLES30.GL_ARRAY_BUFFER, vertexData.length * 4, vertexBuffer, GLES30.GL_STATIC_DRAW);
    GLES30.glEnableVertexAttribArray(a_Position);
    GLES30.glVertexAttribPointer(a_Position, 2, GLES30.GL_FLOAT, false, 8, 0);
    GLES30.glBindBuffer(GLES30.GL_ARRAY_BUFFER, textureVBO);
    GLES30.glBufferData(GLES30.GL_ARRAY_BUFFER, textureData.length * 4, textureBuffer, GLES30.GL_STATIC_DRAW);
    GLES30.glEnableVertexAttribArray(a_TexCoord);
    GLES30.glVertexAttribPointer(a_TexCoord, 2, GLES30.GL_FLOAT, false, 8, 0);
    //这一段到上面都没有看懂？？？
    
//绘制开始
    //激活一个纹理工作区域？：GLES30.GL_TEXTURE0
    GLES30.glActiveTexture(GLES30.GL_TEXTURE0);
    //绑定当前工作纹理为OEXTexture id到纹理工作区域：GLES30.GL_TEXTURE0
    GLES30.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, oesTextureId);
    //设置shader中的纹理采样器，采样当前OESTexture的纹理
    GLES30.glUniform1i(s_Texture, 0);
    //提交draw命令，GPU正式工作。
    GLES30.glDrawArrays(GLES30.GL_TRIANGLE_STRIP, 0, 4);
//绘制结束
    
    //绘制完成，解绑顶点，纹理信息
    GLES30.glDisableVertexAttribArray(a_Position);
    GLES30.glDisableVertexAttribArray(a_TexCoord);
    //解绑OEXTexture
    GLES30.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0);
    //解绑Unity texture于FBO
    GLES30.glFramebufferTexture2D(GLES30.GL_FRAMEBUFFER, GLES30.GL_COLOR_ATTACHMENT0, GLES30.GL_TEXTURE_2D, 0, 0);
    //解绑FBO
    GLES30.glBindFramebuffer(GLES30.GL_FRAMEBUFFER, 0);
}
  • 实际使用中，这一段完成的事情为
    • 启用FBO，绑定unity texture到FBO。
    • 把OESTexture采样到FBO。
    • 关闭FBO，完成所有的操作。

小结

  • 整个过程其实是OESTexture -> 2DTexture的一次采样（复制），其中Surface的数据是填充在内存的，而OESTexture往2DTexture的采样是由GPU完成的。

  • 坑点和难点：

    • 出现了闪屏的问题：

      • 1
        2
        //激活一个纹理工作区域？：GLES30.GL_TEXTURE0
        GLES30.glActiveTexture(GLES30.GL_TEXTURE0);

      • 刚开始的时候出现了闪屏的问题，检查下来发现可能在Unity侧激活的纹理工作区也是TEXTURE0，这样会导致两边同时用到这块工作区域，发生了闪屏。

    • OESTexture属于EXT的范畴，因此在fragment shader中需要标识："#extension GL_OES_EGL_image_external : require"

    • Unity 2DTexture为什么不能直接使用？

      • 参考SurfaceTexture的注释：

      • The texture object uses the GL_TEXTURE_EXTERNAL_OES texture target

额外收获

dump texture

在调试的过程中，使用renderDoc以及高通的SnapDragon Profiler都出现了kernel panic的问题，远程办公的情况下需要call help让同事去工位上长按电源键重启手机，所以比较无奈只能去尝试软件dump texture，好在最后是找到了。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    public void saveTextureToImageWithType(int texture_id, int width, int height, boolean isUnity2DTexture) {
        int[] old_fbo = new int[1];
        GLES30.glGetIntegerv(GLES30.GL_FRAMEBUFFER_BINDING, old_fbo, 0);
        int[] tmp_fbo = new int[1];
        GLES30.glGenFramebuffers(1, tmp_fbo, 0);
        // 根据是否是unity texture来区分attatch到FBO的texture type：
        // unity 		  -> GLES30.GL_TEXTURE_2D
        // SurfaceTexture -> GLES11Ext.GL_TEXTURE_EXTERNAL_OES
        if (isUnity2DTexture) {
            GLES30.glFramebufferTexture2D(GLES30.GL_FRAMEBUFFER, GLES30.GL_COLOR_ATTACHMENT0, GLES30.GL_TEXTURE_2D,
                    texture_id, 0);
        } else {
            GLES30.glFramebufferTexture2D(GLES30.GL_FRAMEBUFFER, GLES30.GL_COLOR_ATTACHMENT0, GLES11Ext.GL_TEXTURE_EXTERNAL_OES,
                    texture_id, 0);
        }
        
        ByteBuffer outRgbaBuf=ByteBuffer.allocate(width*height*4);
        //实际看下来，这个函数只可读取到FBO的数据
        GLES30.glReadPixels(0, 0, width, height, GLES30.GL_RGBA, GLES30.GL_UNSIGNED_BYTE, outRgbaBuf);
        //由于我在调试中会直接打断点查看bitmap的样子，所以这边直接recycle了，实际使用中可以save file
        Bitmap bmp = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
        bmp.copyPixelsFromBuffer(outRgbaBuf);
        bmp.recycle();
		
        //把原来的fbo绑定到framebuffer
        GLES30.glBindFramebuffer(GLES30.GL_FRAMEBUFFER, old_fbo[0]);
        GLES30.glDeleteFramebuffers(1, tmp_fbo, 0); }
}

RenderDoc

实际使用中发现RenderDoc也是比较好用的：https://renderdoc.org/

后续做GPU相关工作的时候也可以用上，高通的平台虽然高端，但是也不能老是依赖于高通的ToolChain。